{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实战演示\n",
    "\n",
    "#### 回顾一下：\n",
    "\n",
    "上网的全过程:\n",
    "\n",
    "- 普通用户\n",
    "\n",
    "打开浏览器 --> 往目标站点发送请求 --> 接收响应数据 --> 渲染到页面上。\n",
    "\n",
    "- 爬虫程序\n",
    "\n",
    "模拟浏览器 --> 往目标站点发送请求 --> 接收响应数据 --> 提取有用的数据 --> 保存到本地/数据库。\n",
    "\n",
    "\n",
    "爬虫的过程：\n",
    "\n",
    "1. 发送请求（requests模块）\n",
    "\n",
    "2. 获取响应数据（服务器返回）\n",
    "\n",
    "3. 解析并提取数据（re正则）\n",
    "\n",
    "4. 保存数据\n",
    "\n",
    "\n",
    "### 网易云音乐\n",
    "\n",
    "爬取网易云音乐，我们将会通过代码爬取歌词并写入到本地\n",
    "\n",
    "来尝试收集下面这首歌的歌词\n",
    "https://music.163.com/#/song?id=191232"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "网站是： http://music.163.com/api/song/lyric?id=191232&lv=1&kv=1&tv=-1\n",
      "{\"sgc\":false,\"sfy\":false,\"qfy\":false,\"lyricUser\":{\"id\":2845491,\"status\":99,\"demand\":0,\"userid\":547896341,\"nickname\":\"UFoundMonster\",\"uptime\":1577327678701},\"lrc\":{\"version\":62,\"lyric\":\"[00:00.000] 作词 : 潘源良\\n[00:01.000] 作曲 : 谷村新司\\n[00:02.000]编曲 : 卢东尼\\n[00:03.000]监制 : 欧丁玉\\n[00:04.000]Publisher : Noel Ms. Publ. (Spraylot)\\n[00:05.000]\\n[00:25.151]让晚风轻轻吹送了落霞\\n[00:30.632]我已习惯每个傍晚去想她\\n[00:36.859]在远方的她 此刻可知道\\n[00:43.057]这段情在我心始终记挂\\n[00:49.290]在这半山\\n[00:50.670]那天我知我知快将要别离没说话\\n[00:55.507]望向她\\n[00:56.853]却听到她说不要相约\\n[00:58.586]纵使分隔 相爱不会害怕\\n[01:01.577]遥遥万里 心声有否偏差\\n[01:08.023]正是让这爱试出真与假\\n[01:14.075]遥远的她\\n[01:16.186]仿佛借风声跟我话\\n[01:20.476]热情若没变 哪管它沧桑变化\\n[01:28.945]\\n[01:39.192]但这天收到她爸爸的一封信\\n[01:45.418]信里面说血癌已带走她\\n[01:51.680]但觉得空虚的心\\n[01:54.327]仿佛已僵化\\n[01:57.890]过去事像炮弹 心中爆炸\\n[02:04.069]在这半山\\n[02:05.452]这天我悲痛悲痛不已\\n[02:07.527]在胡乱说话\\n[02:10.350]夜雨中似听到\\n[02:12.248]她说不要相约\\n[02:13.373]纵使分隔 相爱不会害怕\\n[02:16.304]人无觅处\\n[02:18.403]心声有否偏差\\n[02:22.832]正是让这爱试出真与假\\n[02:28.626]遥远的她\\n[02:30.978]仿佛借风声跟我话\\n[02:35.272]热情若冇变 哪管它沧桑变化\\n[02:43.834]\\n[03:06.057]遥远的她 不可以再归家\\n[03:12.655]我在梦里却始终只有她\\n[03:18.522]遥远的她\\n[03:20.853]可知我心中的说话\\n[03:25.173]热情并冇变 哪管它沧桑变化\\n[03:30.923]遥远的她 不可以再归家\\n[03:37.667]我在梦里却始终只有她\\n[03:43.692]遥远的她\\n[03:45.766]可知我心中的说话\\n[03:50.156]热情并冇变 哪管它沧桑变化\\n[04:02.930]\\n\"},\"klyric\":{\"version\":34,\"lyric\":\"[ti:遥远的她]\\n[ar:张学友]\\n[by:8024771@网易云音乐]\\n[#:http://music.163.com/#/song?id=191232]\\n[25300,5841](0,660)让(0,260)晚(0,626)风(0,250)轻(0,184)轻(0,316)吹(0,459)送(0,411)了(0,375)落(0,2300)霞\\n[30355,6642](0,584)我(0,232)已(0,665)习(0,765)惯(0,215)每(0,350)个(0,375)傍(0,400)晚(0,215)去(0,570)想(0,2271)她\\n[36389,6468](0,558)在(0,336)远(0,394)方(0,240)的(0,778)她(0,227)此(0,435)刻(0,506)可(0,744)知(0,2250)道\\n[43306,6627](0,525)这(0,212)段(0,320)情(0,550)在(0,505)我(0,1060)心(0,251)始(0,404)终(0,460)记(0,2340)挂\\n[49339,6498](0,538)在(0,362)这(0,225)半(0,476)山(0,194)那(0,565)天(0,265)我(0,175)知(0,238)我(0,242)知(0,185)快(0,205)将(0,185)要(0,240)别(0,190)离(0,265)没(0,320)说(0,1628)话\\n[55359,3452](0,635)望(0,395)向(0,635)她(0,205)却(0,193)听(0,197)到(0,145)她(0,245)说(0,200)不(0,210)要(0,230)相(0,162)约\\n[58374,3243](0,553)纵(0,195)使(0,190)分(0,190)隔(0,240)相(0,220)爱(0,220)不(0,205)会(0,200)害(0,1030)怕\\n[61369,6669](0,510)遥(0,350)遥(0,385)万(0,1200)里(0,245)心(0,445)声(0,450)有(0,402)否(0,462)偏(0,2220)差\\n[68306,6365](0,644)正(0,471)是(0,310)让(0,214)这(0,1036)爱(0,251)试(0,401)出(0,473)真(0,445)与(0,2120)假\\n[74313,6761](0,545)遥(0,430)远(0,460)的(0,1141)她(0,233)仿(0,431)佛(0,315)借(0,494)风(0,440)声(0,311)跟(0,309)我(0,1652)话\\n[80359,9203](0,521)热(0,429)情(0,269)若(0,241)没(0,875)变(0,250)那(0,195)管(0,560)它(0,397)沧(0,663)桑(0,401)变(0,4402)化\\n[99341,6456](0,650)但(0,325)这(0,455)天(0,225)收(0,495)到(0,445)她(0,194)爸(0,516)爸(0,724)的(0,311)一(0,394)封(0,1722)信\\n[105357,6550](0,605)信(0,265)里(0,495)面(0,820)说(0,250)血(0,490)癌(0,460)已(0,220)带(0,635)走(0,2310)她\\n[111382,6617](0,595)但(0,341)觉(0,495)得(0,204)空(0,495)虚(0,555)的(0,255)心(0,245)仿(0,1032)佛(0,388)已(0,320)僵(0,1692)化\\n[118315,6626](0,553)过(0,388)去(0,240)事(0,365)象(0,560)炮(0,836)弹(0,354)心(0,335)中(0,475)爆(0,2520)炸\\n[124336,6740](0,602)在(0,446)这(0,156)半(0,471)山(0,193)这(0,624)天(0,205)我(0,220)悲(0,155)痛(0,155)悲(0,150)痛(0,253)不(0,220)已(0,260)在(0,198)胡(0,209)乱(0,235)说(0,1988)话\\n[130380,3307](0,595)夜(0,352)雨(0,623)中(0,228)似(0,162)听(0,195)到(0,233)她(0,142)说(0,155)不(0,193)要(0,157)相(0,272)约\\n[133381,3155](0,535)纵(0,185)使(0,195)分(0,205)隔(0,223)相(0,192)爱(0,269)不(0,189)会(0,212)害(0,950)怕\\n[136367,6740](0,518)人(0,282)无(0,454)觅(0,1229)处(0,312)心(0,440)声(0,200)有(0,610)否(0,505)偏(0,2190)差\\n[143311,6305](0,600)正(0,460)是(0,208)让(0,286)这(0,1056)爱(0,250)试(0,505)出(0,555)真(0,625)与(0,1760)假\\n[149310,6917](0,655)遥(0,415)远(0,420)的(0,1140)她(0,350)仿(0,435)佛(0,255)借(0,575)风(0,415)声(0,285)跟(0,240)我(0,1732)话\\n[155372,9047](0,550)热(0,400)情(0,215)若(0,200)没(0,868)变(0,238)那(0,254)管(0,575)它(0,260)沧(0,985)桑(0,390)变(0,4112)化\\n[186366,6800](0,641)遥(0,399)远(0,355)的(0,1121)她(0,299)不(0,240)可(0,495)以(0,505)再(0,435)归(0,2310)家\\n[193316,6300](0,523)我(0,412)在(0,300)梦(0,199)里(0,1116)却(0,255)始(0,459)终(0,500)只(0,416)有(0,2120)她\\n[199317,6737](0,670)遥(0,339)远(0,411)的(0,1090)她(0,230)可(0,490)知(0,440)我(0,375)心(0,435)中(0,260)的(0,305)说(0,1692)话\\n[205361,6164](0,545)热(0,440)情(0,220)并(0,210)没(0,967)变(0,252)那(0,227)管(0,554)它(0,410)沧(0,798)桑(0,509)变(0,1032)化\\n[211348,6859](0,690)遥(0,423)远(0,437)的(0,1183)她(0,279)不(0,419)可(0,373)以(0,495)再(0,440)归(0,2120)家\\n[218304,6420](0,555)我(0,445)在(0,235)梦(0,245)里(0,1255)却(0,276)始(0,419)终(0,490)只(0,440)有(0,2060)她\\n[224316,6856](0,613)遥(0,426)远(0,354)的(0,1172)她(0,239)可(0,310)知(0,451)我(0,424)心(0,440)中(0,376)的(0,259)说(0,1792)话\\n[230372,6483](0,555)热(0,430)情(0,255)并(0,186)没(0,940)变(0,244)那(0,190)管(0,506)它(0,434)沧(0,812)桑(0,499)变(0,1432)化\\n\"},\"tlyric\":{\"version\":0,\"lyric\":\"\"},\"code\":200}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json # 后面学习\n",
    "import re   # 后面学习,正则表达\n",
    "lrc_url = 'http://music.163.com/api/song/lyric?' + 'id=' + str(191232) + '&lv=1&kv=1&tv=-1'\n",
    "print(\"网站是：\",lrc_url)\n",
    "headers = {\"user-agent\": \"Mozilla/5.0 (Linux; Android 8.1.0; ALP-AL00 Build/HUAWEIALP-AL00; wv)\"}\n",
    "# lyric = requests.get(lrc_url) # 没有标题信息，返回结果不一样，网站会发现你是爬虫就不返回结果给你了。\n",
    "lyric = requests.get(lrc_url, headers=headers)\n",
    "print(lyric.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是发现里面还多了一些不需要的东西。\n",
    "\n",
    "第二步，筛选出我们需要的元素，这里我们将用到正则表达式和json，切记用之前先导入这两个模块。加入几行代码，帅选后的结果为："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "作词 : 潘源良\n",
      " 作曲 : 谷村新司\n",
      "编曲 : 卢东尼\n",
      "监制 : 欧丁玉\n",
      "Publisher : Noel Ms. Publ. (Spraylot)\n",
      "\n",
      "让晚风轻轻吹送了落霞\n",
      "我已习惯每个傍晚去想她\n",
      "在远方的她 此刻可知道\n",
      "这段情在我心始终记挂\n",
      "在这半山\n",
      "那天我知我知快将要别离没说话\n",
      "望向她\n",
      "却听到她说不要相约\n",
      "纵使分隔 相爱不会害怕\n",
      "遥遥万里 心声有否偏差\n",
      "正是让这爱试出真与假\n",
      "遥远的她\n",
      "仿佛借风声跟我话\n",
      "热情若没变 哪管它沧桑变化\n",
      "\n",
      "但这天收到她爸爸的一封信\n",
      "信里面说血癌已带走她\n",
      "但觉得空虚的心\n",
      "仿佛已僵化\n",
      "过去事像炮弹 心中爆炸\n",
      "在这半山\n",
      "这天我悲痛悲痛不已\n",
      "在胡乱说话\n",
      "夜雨中似听到\n",
      "她说不要相约\n",
      "纵使分隔 相爱不会害怕\n",
      "人无觅处\n",
      "心声有否偏差\n",
      "正是让这爱试出真与假\n",
      "遥远的她\n",
      "仿佛借风声跟我话\n",
      "热情若冇变 哪管它沧桑变化\n",
      "\n",
      "遥远的她 不可以再归家\n",
      "我在梦里却始终只有她\n",
      "遥远的她\n",
      "可知我心中的说话\n",
      "热情并冇变 哪管它沧桑变化\n",
      "遥远的她 不可以再归家\n",
      "我在梦里却始终只有她\n",
      "遥远的她\n",
      "可知我心中的说话\n",
      "热情并冇变 哪管它沧桑变化\n"
     ]
    }
   ],
   "source": [
    "json_obj = lyric.text\n",
    "j = json.loads(json_obj)\n",
    "lrc = j['lrc']['lyric']\n",
    "pat = re.compile(r'\\[.*\\]')\n",
    "lrc = re.sub(pat, \"\", lrc)\n",
    "lrc = lrc.strip()\n",
    "print(lrc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 最后一步：保存数据\n",
    "\n",
    "把爬到的歌词放到文本中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_file_path = \"lrc.txt\"\n",
    "with open(new_file_path, 'w') as my_file:  # 这些我们后面一个个详细介绍\n",
    "    my_file.write(lrc)\n",
    "\n",
    "#运行之后，就会发现多了一个文档，打开就看到，歌词已经在里面。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JSON 函数\n",
    "\n",
    "使用 JSON 函数需要导入 json 库：import json。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "|函数|描述|\n",
    "|-|-|\n",
    "|json.dumps |将 Python 对象编码成 JSON 字符串\n",
    "|json.loads\t|将已编码的 JSON 字符串解码为 Python 对象\n",
    "\n",
    "\n",
    "### 编码--json.dumps\n",
    "\n",
    "json.dumps 用于将 Python 对象编码成 JSON 字符串。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "data = {'b': True, \"a\": 1, 'nested': {'foo': 'bar'}, 'c': None, 'some_list': [1, 2, 3]}\n",
    "print('编码前类型: {} 数据: {}'.format(type(data), data))\n",
    "json_data = json.dumps(data)\n",
    "print('编码后类型: {} 数据: {}'.format(type(json_data), json_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 解码--json.loads\n",
    "\n",
    "json.loads 用于解码 JSON 数据。该函数返回 Python 字段的数据类型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoded = json.loads(json_data)\n",
    "print('解码后type: {} data: {}'.format(type(decoded), decoded))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习1\n",
    "\n",
    "获取网页的其他信息, 歌词贡献者是谁？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_obj = '{\"sgc\":false,\"sfy\":false,\"qfy\":false,\"lyricUser\":{\"id\":2845491,\"status\":99,\"demand\":0,\"userid\":547896341,\"nickname\":\"UFoundMonster\",\"uptime\":1577327678701}}'\n",
    "print(json_obj)\n",
    "j_data = json.loads(json_obj)\n",
    "print(\"歌词贡献者：\", j_data['lyricUser']['nickname'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 文件操作\n",
    "\n",
    "open() 方法\n",
    "Python open() 方法用于打开一个文件，并返回文件对象，在对文件进行处理过程都需要使用到这个函数，如果该文件无法被打开，会抛出 OSError。\n",
    "\n",
    "注意：使用 open() 方法一定要保证关闭文件对象，即调用 close() 方法。\n",
    "\n",
    "open() 函数常用形式是接收两个参数：文件名(file)和模式(mode)。\n",
    "\n",
    "刚才我们用了\"w\"写的模式\n",
    "\n",
    "```\n",
    "new_file_path = \"lrc.txt\"\n",
    "with open(new_file_path, 'w') as my_file:  # 这些我们后面一个个详细介绍\n",
    "    my_file.write(lrc)\n",
    "```\n",
    "\n",
    "下面我们看一下读取文档的例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_path = \"lrc.txt\" # 读取文件的路径名称\n",
    "\n",
    "with open(new_file_path, 'r') as my_file:  # 刚才我们写入歌词的文本\n",
    "    print(my_file.read())              # 读取所有"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_path = \"lrc.txt\"\n",
    "# 我们也可以还原成一行一行地输出 \n",
    "with open(new_file_path, 'r') as my_file: \n",
    "    line_number = 1\n",
    "    for line in my_file.readlines():  # 一行一行地读取\n",
    "        print(f\"{line_number:02}:\", end=\" \")\n",
    "        line_number += 1\n",
    "        print(line.strip()) # 把末尾的'\\n'删掉"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 常见文档操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "current_file = os.path.realpath('scrapy_2_example.ipynb')  \n",
    "print('查看当前文件路径名称: {}'.format(current_file))\n",
    "# Note: in .py files you can get the path of current file by __file__\n",
    "\n",
    "current_dir = os.path.dirname(current_file)  \n",
    "print('查看当成目录: {}'.format(current_dir))\n",
    "# Note: in .py files you can get the dir of current file by os.path.dirname(__file__)\n",
    "\n",
    "data_dir = os.path.join(os.path.dirname(current_dir), 'food_store')\n",
    "print('自定义路径: {}'.format(data_dir))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 要列出当前目录下的所有目录\n",
    "for x in os.listdir('.'):\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python 正则表达式\n",
    "\n",
    "正则表达式是一个特殊的字符序列，它能帮助你方便的检查一个字符串是否与某种模式匹配。\n",
    "\n",
    "Python 自1.5版本起增加了re 模块，它提供 Perl 风格的正则表达式模式。\n",
    "\n",
    "re 模块使 Python 语言拥有全部的正则表达式功能。\n",
    "\n",
    "compile 函数根据一个模式字符串和可选的标志参数生成一个正则表达式对象。该对象拥有一系列方法用于正则表达式匹配和替换。\n",
    "\n",
    "re 模块也提供了与这些方法功能完全一致的函数，这些函数使用一个模式字符串做为它们的第一个参数。\n",
    "\n",
    "我们使用了comile和sub, 我们一起看看怎么使用\n",
    "\n",
    "```\n",
    "pat = re.compile(r'\\[.*\\]')  # compile\n",
    "lrc = re.sub(pat, \"\", lrc)   # sub\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### re.compile 函数\n",
    "compile 函数用于编译正则表达式，生成一个正则表达式（ Pattern ）对象，供 match() 和 search() 这两个函数使用。\n",
    "\n",
    "语法格式为：\n",
    "\n",
    "```re.compile(pattern[, flags])```\n",
    "\n",
    "- pattern : 一个字符串形式的正则表达式\n",
    "\n",
    "- flags : 可选，表示匹配模式，比如忽略大小写，多行模式等，具体参数为：\n",
    "\n",
    "- re.I 忽略大小写\n",
    "- re.L 表示特殊字符集 \\w, \\W, \\b, \\B, \\s, \\S 依赖于当前环境\n",
    "- re.M 多行模式\n",
    "- re.S 即为 . 并且包括换行符在内的任意字符（. 不包括换行符）\n",
    "- re.U 表示特殊字符集 \\w, \\W, \\b, \\B, \\d, \\D, \\s, \\S 依赖于 Unicode 字符属性数据库\n",
    "- re.X 为了增加可读性，忽略空格和 # 后面的注释"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pattern = re.compile(r'\\[.*\\]')\n",
    "m = pattern.match('[01:20.476]热情若没变 哪管它沧桑变化')\n",
    "print(m)\n",
    "print(m.group(0))  # 返回匹配成功的整个子串\n",
    "print(m.start(0), m.end(0), m.span())  # 返回匹配成功的位置信息"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 检索和替换\n",
    "\n",
    "Python 的 re 模块提供了re.sub用于替换字符串中的匹配项。\n",
    "\n",
    "语法：\n",
    "\n",
    "```re.sub(pattern, repl, string, count=0, flags=0)```\n",
    "\n",
    "- pattern : 正则中的模式字符串。\n",
    "- repl : 替换的字符串，也可为一个函数。\n",
    "- string : 要被查找替换的原始字符串。\n",
    "- count : 模式匹配后替换的最大次数，默认 0 表示替换所有的匹配。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = '[01:20.476]热情若没变 哪管它沧桑变化'\n",
    "pattern = re.compile(r'\\[.*\\]')\n",
    "new_text = re.sub(pattern, \"!!\", text)  # 这里做了两步，第一步把[01:20.476]找出来，然后用xx来替换\n",
    "print(new_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phone = \"0757-86547-1548 # 001789\"\n",
    " \n",
    "# 删除字符串中的 Python注释 \n",
    "num = re.sub(r'#.*$', \"\", phone)  # 简单的正则也可以直接写，不需要compile去包装\n",
    "print(\"第一步，电话号码是:\", num)\n",
    " \n",
    "# 删除非数字(-)的字符串 \n",
    "# num = re.sub(r'\\D', \"\", phone)  # 这个也可以，+D意思是非数字的字符，就是把非数字字符变成空\n",
    "num = re.sub(r'-', \"\", num)   # 把-替换为空\n",
    "print(\"第二步，电话号码是:\", num)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习2 \n",
    "\n",
    "网路文明用语规范越来越重视，用户留言很多字会被替换掉，下面我们用正则把语句中所有“傻逼”字变成“sb”。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"你是个大傻逼，知道吗？真是个傻逼！\"\n",
    "\n",
    "pattern = re.compile(r'傻逼')\n",
    "new_text = re.sub(pattern, \"sb\", text)\n",
    "print(\"变成文明语句: \", new_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习3\n",
    "\n",
    "从网页中获取歌曲的id，歌曲的名称以及歌词的网页ID。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json \n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 没有网络可以用这个\n",
    "file_path = \"localhost/song_list.html\" # 读取文件的路径名称\n",
    "song_list = \"\"\n",
    "with open(file_path, 'r') as my_file:  # 刚才我们写入歌词的文本\n",
    "    song_list=my_file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 原来网站：https://music.163.com/#/artist?id=6460\n",
    "# 自己本地启动服务器，可以用这个\n",
    "url = 'http://127.0.0.1:8000/song_list.html'\n",
    "print(\"网站是：\",url)\n",
    "headers = {\"user-agent\": \"Mozilla/5.0 (Linux; Android 8.1.0; ALP-AL00 Build/HUAWEIALP-AL00; wv)\"}\n",
    "result = requests.get(url)\n",
    "result.encoding='gbk'  # 如果有乱码，换一个编码\n",
    "song_list = result.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# line = '<a href=\"/album?id=3084625\" title=\"醒着做梦\">' # 测试使用\n",
    "pattern = re.compile(r'album\\?id=(.*?)\" title=\"(.*?)\"')\n",
    "m = pattern.findall(song_list)\n",
    "print(m)  # 输出歌单"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pattern = re.compile(r'song\\?id=(.*?)\"')\n",
    "lrc_ids = pattern.findall(song_list)\n",
    "print(lrc_ids) # 输出歌词页面ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
